Skip to content

fix(agc): mirror only unimported tails of registered heaps - #8380

Merged
adriadam10 merged 2 commits into
boykopovar:mainfrom
Zenodeon:agc-budgeted-host-imports
Oct 24, 2026
Merged

adriadam10 merged 2 commits into
boykopovar:mainfrom
Zenodeon:agc-budgeted-host-imports

Conversation

@Zenodeon

@Zenodeon Zenodeon commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

What

When a registered heap exceeds the host-import budget, import the fitting prefix and mirror only its unimported tail. Previously a budget refusal could mirror the whole registration, including bytes already represented by an import. Keep the original registration identity for lifetime tracking while keying and allocating each mirror by its selected physical range.

Rebased onto main ea695358ee1ccaf3d108b720d82c5d88413d43dc, dropping the merged #7292 prerequisite. This PR contains the standalone resource fix and a focused test follow-up. The partial mirror uses main's #7647 reservation and sparse-resident accounting: reserveHeapBytes, ResidentBuffer::Create, accounted, listed and unlistMirror, including reservation rollback and retirement. Removed the extra blank lines after importPieces.

Related work: #9514 changes budget-refusal eviction before the address-copy cap. Agreed merge order is #9514 first, then rebase this PR onto the resulting main. Its implementation is excluded from this branch. The local combined build needs one bookkeeping adaptation in #9514's added retry path: replace imported.push_back(range->address) with imported.emplace_back(range->address, range->address + range->bytes) to match this PR's imported-range tracking.

Tested

  • Windows 11 Pro 10.0.26200; NVIDIA RTX 4070, driver 610.88; GCC 15.2, Release, SPIR-V Tools enabled. Rebuilt complete libs and recorded local library hashes. No game or shader patches included.
  • Same 16 focused tests on clean main ea695358ee1ccaf3d108b720d82c5d88413d43dc and candidate 41d4affd6191d6b799f838a8e56b216335b563a2: 16/16 passed on each platform (Windows RTX 4070 and Ubuntu 26.04.1 under WSL2, lavapipe / llvmpipe LLVM 21.1.8, Mesa 26.0.8). Linux main was rerun during this follow-up; Windows main results are from the unchanged pinned base's prior matched run. Used ANYPS5_REQUIRE_VULKAN=1, ANYPS5_NO_SHADER_CACHE=1, VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation and verbose CTest with a 30-second individual timeout. Verified the layer actually loaded; all four logs have no missing-layer warnings, skipped tests or VUID- errors.
  • Coverage: graphics/BDA device and span reads, real GPU resident heap mirrors, CPU device selection, draw-pipeline span reads, copied/sparse mirror unwatch with and without space caching, sparse budget, run unwatch, address-space teardown and specialized BDA shader ABI none/full.
  • Prior head 57851797857808c16b0380ba08f8df118e02dc36, composed locally with fix(agc): an address-based build evicts idle imports instead of copying past the cap #9514 head f2e4290e9c75880fc964f56528414713d2e21e96 and the imported-range bookkeeping adaptation above: 18/18 passed, adding agc_address_copy_cap and agc_address_copy_cap_remembered. Actual validation layer loaded, no skipped tests, missing-layer warnings or VUID- errors. This composition predates the test-only follow-up and remains local.
  • Reproduced the old CI failure locally on lavapipe: read heap 0x769c65ff0000 had no imported prefix; the lower-addressed write heap 0x769c657e0000 had the 131072-byte prefix. The test incorrectly assumed allocation order equaled registry address order. This was not a sparse-block granularity difference. The follow-up verifies the exact prefix on the selected heap and tests both allocation orders with resident and copied fallback mirrors.
  • Negative control: the updated real GPU regression over main fails the first registered heap has no exact two-block imported prefix. The earlier Windows-only mocked cases also fail on main (a heap exceeding the import budget did not import its fitting prefix). No Vulkan validation errors in the negative controls.
  • partialHeapImportBudgetTests covers a 384 KiB registration with a 256 KiB imported prefix and exact 128 KiB tail, mirror replacement, first/last bytes, CPU writes, GPU write-back, short final windows, neighbors and retirement. Added the simultaneous import-window and budget-limit case.
  • Real GPU regression exercises sparse-backed and copied fallback mirrors for an 8 MiB registration with an asserted 128 KiB imported prefix. Its sparse tail accounts for 64 KiB when the selected heap is the read heap, or 128 KiB when it is the write heap (tail data plus the GPU-written block). Copied fallback accounts for the 8 MiB minus the verified prefix. GPU reads and subsequent CPU edits preserve exact values and neighboring bytes. These are mirror accounting results, not total physical VRAM, process RAM or FPS measurements; allocator slabs may remain allocated.
  • Gameplay, cross-title and extended stability checks have not been rerun on this revision. No FPS or process-memory improvement is claimed.
  • git diff --check and tools/check_conventions.py --base upstream/main pass. The required --base origin/main invocation reports inherited findings against the older fork main; those findings are outside this topic diff.

Checklist

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request Conflicting files (lines on main) Same exports Same new files
PR 8963 core/libs/prx/libSceAgcDriver/tests/BdaResources.cpp L2191
PR 9551 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp L2863, L2869-2876, L3870, L3898, L3901-3903, L3905, L3913

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 1191 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 2434 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 4448 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
core/libs/prx/libSceAgcDriver/tests/BdaResources.cpp
PR 5021 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
core/libs/prx/libSceAgcDriver/tests/BdaResources.cpp
PR 6186 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 6197 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
core/libs/prx/libSceAgcDriver/tests/BdaResources.cpp
PR 6412 core/libs/prx/libSceAgcDriver/tests/BdaResources.cpp
PR 6481 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 6548 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 6628 core/libs/prx/libSceAgcDriver/tests/BdaResources.cpp
PR 6918 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
core/libs/prx/libSceAgcDriver/tests/BdaResources.cpp
PR 7201 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
core/libs/prx/libSceAgcDriver/tests/BdaResources.cpp
PR 7457 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 7642 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
core/libs/prx/libSceAgcDriver/tests/BdaResources.cpp
PR 9508 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
core/libs/prx/libSceAgcDriver/tests/BdaResources.cpp
PR 9879 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
core/libs/prx/libSceAgcDriver/tests/BdaResources.cpp
PR 9898 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 9940 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 9941 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 10193 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
core/libs/prx/libSceAgcDriver/tests/BdaResources.cpp
PR 10194 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp

@github-actions github-actions Bot added the agc libSceAgc* label Oct 20, 2026
@Zenodeon
Zenodeon marked this pull request as ready for review October 20, 2026 13:46
@TheSovietWaffle

Copy link
Copy Markdown
Contributor

AMD data point: RTX 3080 (NVIDIA 617.42) and AMD Radeon RDNA 2 iGPU (1002:164e, 32.0.21045.5002), Windows 10 Pro 22H2, WinLibs GCC 15.2.0, Release, head 9a20ec2b8: agc_driver_graphics (which includes the new BdaResources.cpp cases) passes 3 of 3 on both GPUs.

AI-assisted: yes (Claude Code).

@EagleBearerYS

Copy link
Copy Markdown
Contributor

@Zenodeon a merge-order note: #9514 changes the plain import in AcquireRegistered, at the spot where this PR answers a refused import with budgeted pieces. When the host import budget refuses a range, #9514 retries the import with eviction, and only when copying the range would take the build past APS5_ADDRESS_COPY_MAX_MIB. A range your pieces serve isn't copied, so #9514's retry doesn't run for it, and the two should compose. As this one depends on #7292, a draft, #9514 proposes going first, and this PR would rebase over it. If you'd rather this one went first, say so and #9514 will list it under Depends on.

@MarcZX10

Copy link
Copy Markdown
Contributor

I am reviewing #9514, which changes AcquireRegistered at the import site adjacent to #8380’s piece-import branch. Its author proposes that #9514 merge first, then #8380 rebase; the stated rationale is that piece-served ranges do not take the budget-refusal retry. Please confirm whether that order is safe for #8380, or describe the needed sequence, so both PRs can record the same agreed dependency/order before merge.

@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 23, 2026
@adriadam10 adriadam10 self-assigned this Oct 23, 2026

@adriadam10 adriadam10 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#7292 is merged, so the first commit can be dropped; the branch needs a rebase onto current main, and it is not mechanical:

  • GuestBufferMemory.cpp acquireMirror conflicts in four places. main now accounts heap mirrors through unlistMirror, reserveHeapBytes, ResidentBuffer::Create and mirror->accounted (#7647); this PR still uses heapMirrorsBeside(selected) and state.heapBytes +=/-=. The partial selected range (address/bytes) has to go through the new reservation and error paths (heapMirrorError(range->address, range->bytes, held, ...), mirror->listed).
  • After rebasing, re-run the new partialHeapImportBudgetTests with ANYPS5_REQUIRE_VULKAN=1, ANYPS5_NO_SHADER_CACHE=1, validation layer, and check the MirrorCounters().heapBytes assertions still hold with resident (sparse-backed) mirrors.
  • GuestBufferMemory.cpp:1694-1695: two blank lines after importPieces.
  • #9514 touches the same import site in AcquireRegistered; @billythemenace2 confirmed it composes if #9514 lands first, so rebase over it if it merges before you push.

@Zenodeon
Zenodeon force-pushed the agc-budgeted-host-imports branch from 9a20ec2 to 5785179 Compare October 23, 2026 19:16
@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 23, 2026

@adriadam10 adriadam10 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The full Build on this head fails agc_driver_resident_heap_mirror on the Linux job (lavapipe, run 40011794592), in the new PartialImportTests:

  • ResidentHeapMirror.cpp:294: partial heap import: its mirror holds 131072 bytes, expected 65536. Log: host import budget: 0 MiB live of 0 MiB ... 1 imports (8 MiB) refused so far. The test is new in this PR, so main has no equivalent failure.
  • On lavapipe the sparse mirror of the tail accounts two blocks where the RTX 4070 run accounts one. Either the imported prefix or the sparse block granularity differs per driver, and the test pins Block for the non-copied case.

Please check which it is: if the code selects a different prefix/tail on lavapipe, fix it there; if only the granularity differs, derive the expected bytes from the imported prefix and the device's sparse block size instead of the fixed Block. The copied case (expected = HeapBytes - 2 * Block) needs the same check.

@Zenodeon

Copy link
Copy Markdown
Contributor Author

Updated in 5785179: rebased onto main ea69535, dropped the merged #7292 prerequisite, routed selected mirror ranges through the current reservation/accounting and cleanup paths, and removed the extra blank lines.

On Windows 11 / RTX 4070 (610.88), the same 16 focused tests passed on main and this head with required Vulkan, shader cache disabled and the validation layer confirmed loaded; neither run added VUID errors. Both new regressions fail on main as expected. A separate local combination with #9514 head f2e4290 passed 18/18 tests after adapting its added imported.push_back to this PR's range-pair bookkeeping. Confirming the proposed order: #9514 first, then rebase #8380 over the resulting main. #9514's implementation is not included here.

Thanks for catching the Linux CI failure in the new PartialImportTests. The Windows results do not resolve that failure: lavapipe reports 131072 accounted bytes while the test expects 65536. The remaining task is to distinguish actual prefix/tail selection from sparse backing granularity and validate the copied case as well; simply loosening the assertion would not establish correctness. This head is not yet validated across both platforms. No gameplay FPS or process-RAM improvement is claimed.

@Zenodeon

Copy link
Copy Markdown
Contributor Author

Fixed in 41d4aff. Reproduced the Linux failure: the lower-addressed write heap received the imported prefix, while the test incorrectly checked the read heap. It was allocation order, not sparse-block granularity.

The regression now verifies the actual prefix and checks both allocation orders with sparse and copied mirrors. Updated head: 16/16 focused tests pass on Windows RTX 4070 and Linux lavapipe, with validation loaded and no VUID errors. Linux main also passes 16/16; the updated regression fails on main as expected. Full upstream CI still needs a rerun.

@adriadam10
adriadam10 merged commit 363e138 into boykopovar:main Oct 24, 2026
11 of 12 checks passed
@adriadam10

Copy link
Copy Markdown
Collaborator

Re-checked against current main: the new changes to GuestBufferMemory.cpp (deferred small imports) are confined to ImportMappedRanges, away from the import and mirror paths this PR changes, and the ranges in the new tests (384 KiB, 8 MiB) are above the 64 KiB deferral threshold. The partial-import tests pin the exact imported prefix and the mirror's accounted bytes for both heap orders with sparse and copied mirrors; merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agc libSceAgc*

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants